Haptic Interaction with 3D Ultrasound Data

The e-Touch sonoTM system allows users to interactively feel and see 3D ultrasound images. This system is currently targeted for use in pre-natal imaging [1]. Parents are able to feel as well as see three-dimensional imagery. We have found that this increases both their understanding and their enjoyment of their child's ultrasound image. This has several important benefits. First of all, it allows physicians to more clearly explain the developmental process, progress and any complications. Second, it helps ease parental stress and anxiety over the progress of their child. Finally, it helps the all-important parent-child bonding process. Over time, the e-Touch sono system will be refined in order to be used for medical diagnosis, surgical planning, and intraoperative procedures. Background Ultrasound is a high-quality, reliable and cost effective medical diagnostic tool. It has been used for medical imaging and diagnosis purposes since the Second World War [2][3]. The first medical ultrasound systems, mirroring their roots in Sonar technology, required that the patient be immersed in water for imaging to occur. They also used "A-mode" presentation of the ultrasound data -blips on an oscilloscope screen. The more familiar "B-mode" presentation of twodimensional gray scaled images followed shortly thereafter. Ultrasonic imaging came into more common use in the clinical environment in the 1960s and 1970s with the introduction of compact hand-held scanners with real-time B-mode imaging capabilities. More recently, medical ultrasound systems capable of generating three-dimensional (i.e., volumetric) data and images have become available. Some ultrasound systems generate the three-dimensional images by analyzing and combining a series of two-dimensional B-mode images gathered using a hand-held scanner outfitted with a position/orientation tracker. Other systems utilize a more sophisticated transducer design to allow three-dimensional data to be gathered without requiring any external transducer movement or tracking. These later systems are often referred to as "4D" systems because they allow threedimensional data and imagery to be gathered in real-time. GE Medical Systems' Voluson 730 ultrasonic imaging system, for example, currently generates 16 three-dimensional images per second [4]. One of the most common uses of ultrasound is in obstetrics and gynecology. Ultrasound has made it possible to non-invasively study pregnancy from beginning to end. Diagnosis of complications of multiple pregnancy, fetal abnormality and placenta previa became possible with the use of ultrasound in the clinical environment. Moreover, ultrasound exams are generally considered safe at the power levels used for diagnosis. In addition to providing physicians with a useful diagnostic tool, ultrasonic imaging allows prospective parents to view their child's development. Figure 1 -Water Immersion Motorized B-mode Scanner Circa 1954 3D Haptic Interaction As part of our research and development, we have explored the use of haptic interaction in the understanding, use and modification of sensor-derived three-dimensional geoscientific [5] (e.g., seismic) and medical (e.g., CAT Scan or MRI) data. 3D ultrasound is particularly interesting because it is a safe and cost effective real-time imaging modality. Adding haptic interaction to 3D ultrasound promises to have many of the same benefits and features that we have seen before. Three-dimensional haptic interaction allows for the direct and unambiguous selection of parts of the ultrasound image for further exploration/analysis, simplified and direct six degree-of-freedom control of imaging tools such as interactive clipping planes and the use of an additional sensory channel or modality (i.e., touch) for understanding the data. These features hold the promise of simplifying and improving the clinical analysis of three-dimensional ultrasonic imagery by providing both a natural and a rich additional sensory interaction mode. Clearly, however, fully exploring and understanding the clinical impact and benefits of adding touch to ultrasound images will take time. In order to help introduce haptic technology to the three-dimensional ultrasound community, we have developed a product, known as e-Touch(TM) sono, which is aimed at providing a more informative and enjoyable experience for parents when pre-natal 3D ultrasound images are taken of their child. Parents are able to feel as well as see three-dimensional imagery. We have found that this increases both their understanding and their enjoyment of their child's ultrasound image. This has several important benefits. First of all, it allows physicians to more clearly explain the developmental process, progress and any complications. Second, it helps ease parental stress and anxiety over the progress of their child. Finally, it helps the all-important parent-child bonding process. This initial focus on parental interaction with ultrasound allows us to introduce haptic technology into the clinical environment in a way that has clear benefits today and allows the medical/diagnostic uses of haptic interaction with 3D ultrasound to be more gradually explored. The e-Touch sono System The e-Touch sono System is a turnkey hardware and software system that allows users to interactively feel and see 3D ultrasonic images. Sono also allows the 3D images to be interactively cleaned-up and exported for the generation of 3D "hardcopy" or imagery. Figure 2 -The e-Touch sono System The system consists of a computer workstation, a graphical display, a 3D touch interface device and the e-Touch sono software. Sono can also be run on a laptop computer system. Figure 3 -3D Ultrasound Acquisition The overall viewing and haptic interaction process begins when a pregnant women gets a 3D ultrasound. The data, in volumetric form, is transferred to the e-Touch sono system. The program automatically cleans up the data and applies default visualization and haptic parameters so that the "skin" surfaces in the data are visible and touchable. The parent or physician can then see and feel the surfaces of the child's face and body. Haptic skin textures are applied to the surfaces to improve the overall haptic experience. As shown in Figure 4, modern 3D ultrasound machines allow highly detailed images to be generated. Allowing parents to interactively control this imagery as well as "feel their baby" significantly adds to the overall educational, enjoyment and bonding aspects of the experience, in addition to easing anxiety about the health of the baby. Figure 4 -Side by Side Images of Pre-Natal Ultrasound & Baby The key features of the sono system are: • Reads data directly generated by GE Voluson 730 4D Ultrasound Systems o Cartesian Kretz V730 volume data file format via network or CDROM o Other data formats to be added • Interactive visual display of 3D ultrasound data o Control color and opacity in real time o Real time rotate, translate & zoom • Allows user to feel the ultrasound data in three dimensions o Surfaces (variable feel) o Volume properties (variable feel) • Allows interactive clean-up of ultrasound data for 3D export in a variety of formats o Volume o Surface o Point • Generates data for 3D "hardcopy" output o Several processes supported • Generates data for 3D "take home" visualization o Several formats supported